Resilience dimensions in health system performance assessments, European Union

Abstract Objective To explore the definition and operationalization of resilience in health system performance assessments in European Union countries. Methods We conducted multiple empirical case study analyses. We identified relevant cases through a literature review from 2014 to 2023 using Google Scholar and through a snowball technique to retrieve additional information. We included only documents that explicitly mentioned resilience in health system performance assessments. We performed a content analysis to identify common patterns in defining resilience. Findings The final sample consisted of six countries: Belgium, Croatia, Czechia, Estonia, Ireland and Italy. Each country adopted a distinct approach to conceptualizing resilience, with countries prioritizing specific aspects based on lessons learnt from the coronavirus disease 2019 (COVID-19) pandemic. Some countries focused on maintaining essential health-care services and protecting vulnerable groups. Other countries prioritized management capacity, staff preparedness, digital health utilization and strengthening of primary health care. Content analysis revealed six resilience definitions derived from the key performance indicators: addressing unmet needs and maintaining outcomes; protecting vulnerable groups; acquiring and using resources; having trained and prepared staff in place; using digital health; and strengthening primary health care. Conclusion Integration of resilience into the health profiles of European Union countries preceded its inclusion in national health system performance assessments, the latter of which became more prominent after the COVID-19 pandemic. Variations in interpretations within health system performance assessments reflect differences in indicators and policy responses.


Introduction
The coronavirus disease 2019 (COVID-19) pandemic posed a considerable challenge for governments, affecting health, the economy and citizens' well-being. 1 The pandemic exposed weaknesses in health systems, such as insufficient workforce capacity 2 and critical care resources. 3This crisis highlighted the necessity for a resilience-centred approach to equip health systems to deal with a wider spectrum of future shocks, 2 particularly given the varying levels of preparedness among countries. 4,5Failing to prepare for a shock may result in costly interventions with lasting repercussions. 3 In some cases, these repercussions can permanently alter the status quo of healthcare systems, creating a legacy of new challenges.Systems measuring performance, such as health system performance assessments, can support governments in evaluating preparedness, shock management and capacity-building for learning and recovery. 6Resilience can be seen as a cross-cutting dimension of the intermediate and ultimate goals of health systems, as well as a factor influencing the performance of the health system. 7lthough broad consensus exists on the need to bolster health system resilience, questions persist about a shared definition and vision. 8One proposal stated that resilience evaluates a system's ability to maintain performance under major stresses. 9The World Bank added other elements by calling for health systems to be alert to threats, responsive to evolving needs, adaptable to minimize disruptions, and capable of postcrisis transformation based on lessons learnt. 10The definition used in our study is the one proposed by the European Union (EU) expert panel on effective ways of investing in health. 11e panel defined resilience as "the capacity of a health system to (a) proactively foresee, (b) absorb, and (c) adapt to shocks and structural changes in a way that allows it to (i) sustain required operations, (ii) resume optimal performance as quickly as possible, (iii) transform its structure and functions to strengthen the system, and (possibly) (iv) reduce its vulnerability to similar shocks and structural changes in the future." 11 Resilience has been measured with different tools such as the United States Agency for International Development (USAID) tool, the World Health Organization (WHO) Joint External Evaluation tool, the Global Health Security Index, or the proposed resilience index, 9,[12][13][14] as well as numerous dashboards set up during the COVID-19 emergency to provide real-time data. 15Some of these tools are based on standalone systems using key informant surveys or ad hoc and temporary surveillance data, with limited use of established health information systems.In contrast to this specialized approach, scholars have suggested introducing resilience into a broader framework for health system performance assessment, 7,9 especially after the 2013-2016 Ebola virus disease outbreak in West Africa. 8,16Such a framework allows resilience to be monitored within a comprehensive assessment of health system performance.Given that the objectives and functions of health system performance assessment may vary over time, this assessment should be flexible and adaptable. 6Moreover, no single universal approach exists that suits every system. 17The health system performance assessment can be seen as a countryowned, participatory process that allows the health system to be assessed as a whole and linked to national health strategies whenever possible. 18,19 Health system performance assessment, European Union of population health, strategic accountability for health system actions needs to be strengthened and policy-makers and other stakeholders should be engaged in articulating health system objectives and priorities.In this way, actions can be harmonized, progress in attainment of goals gauged and informed decisionmaking stimulated. 20,21espite numerous attempts to conceptualize resilience, many efforts have remained at the theoretical level.Therefore, we aimed to investigate how countries are putting resilience into practice by measuring key performance indicators, in effect demonstrating which priority areas are considered essential components of each country's concept of resilience.

Methods
We performed empirical case study analyses to explore how different countries have integrated the concept of resilience into their health system performance assessment frameworks.At the same time, we investigated whether countries have developed any measurable criteria for assessing resilience.
The case study selection process followed a systematic approach.First, we limited the scope of the analysis to the EU Member States and investigated health system performance assessments to countries with available data.We chose to limit our analysis to the EU for several reasons.First, the EU has played a central role in endorsing initiatives on health system performance assessment since 2014, when it established an expert group on health system performance assessment 21 to facilitate knowledge exchange among member states. 22Additionally, the European Commission, through the technical support instrument, 23 has assisted health authorities in different countries in implementing health system performance assessment frameworks tailored to the country.Second, publicly disclosing health system information is standard practice in the EU through initiatives such as the European Commission's biennial country health profiles, which also include resilience measures. 24Thus, a supranational organization already exists that guides the incorporation of resilience into health system performance assessment.Finally, with the EU moving towards a European Health Union, 25 which aims to facilitate health-care delivery across internal EU borders, 26 a culture of coordination is growing within the union, extending partially to the health-care sector, despite member states retaining sovereignty over health care. 26e conducted a literature review using Google Scholar to identify relevant documents drawn from scientific and grey literature sources published from January 2014 to February 2024, with no language restrictions applied.We chose Google Scholar because of its extensive coverage of peer-reviewed articles, books, conference papers and other reports, including official and institutional documents.We selected 2014 as a reference year because it marked the global recognition of resilience in health-care systems after the Ebola virus disease outbreak.In addition, the European Commission emphasized the importance of resilience in its 2014 publication Communication from the Commission on effective, accessible and resilient health systems. 27fter screening the titles and abstracts of the documents, we integrated the initial sample with information drawn from the websites of health ministries, health boards and health agencies.We examined websites and reports of international organizations, with attention to the countries that received support from the European Commission in developing their health system performance assessment. 23We took this precautionary step to prevent the inadvertent omission of these countries as their national authorities may have not yet published any information, possibly due to ongoing programme activities.We identified additional articles and reports using a snowball technique, 27 starting with the references of seminal studies.
We excluded documents that did not focus on assessment of national health system performance; additionally, we excluded publications that examined specific health-care topics without adopting a comprehensive system-wide perspective.We selected only those health system performance assessments where the resilience dimension was reported (14 documents).In the chosen case studies, we performed a content analysis, 28,29 examining all pertinent documents and websites for the countries starting from the first year resilience was explicitly introduced.In our content analysis, we used a deductive approach 30 to measure resilience, based on the definition provided by the EU expert panel on effective ways of investing in health. 11We initially categorized the key performance indicators documented within the health system performance assessments of the countries in line with the four points outlined in the EU expert panel's definition of resilience, namely: (i) sustaining operations; (ii) resuming optimal performance swiftly; (iii) transforming structure and functions to strengthen the system; and (iv) reducing vulnerability to future shocks. 11After expanding on these four points, we were able to better classify the concept of resilience through the way the key performance indicators operationalized resilience.

Results
We retrieved 886 documents with the review of the literature and the snowball procedure.We excluded most of these documents as they considered health system performance assessment only at a theoretical level without analysing country-owned health system performance assessments.The final sample comprised 40 documents, excluding repeated references, websites and platforms (Fig. 1 and Table 1; Albreht et al., unpublished report, 2023). 17, Tabl2 summarizes EU countries' population, economic and health system profiles.We identified 18 countries that had adopted comprehensive systemwide health system performance assessments.Seven countries explicitly included resilience: Belgium, Croatia, Czechia, Estonia, Ireland, Italy and Lithuania.In Italy, as a decentralized system, we identified and considered the health system performance evaluations conducted at the regional level of government.Four countries had EU support in their health system performance assessments.This limited number of countries may be due to the slow reaction and inertia of complex organizations to change. 56able 3 outlines how health system resilience is conceptualized and operationalized into key performance indicators across six of the seven countries.We omitted Lithuania due to the unavailability of documents that could elucidate how resilience had been interpreted.Italy is repeated in the table due to the periodic release of its health system performance assessments.As highlighted in the year column of

Research
frameworks explicitly incorporating the resilience dimension predominantly emerge in updates after 2019, reflecting a trend after the COVID-19 pandemic.
Table 4 summarizes our content analysis, which revealed six resilience definitions derived from the key performance indicators in selected health system performance assessments.These definitions are compared with the EU expert panel's theoretical four-category definition. 11

Address unmet needs and maintain outcomes
Three health system performance assessments incorporated key performance indicators related to the ability to maintain essential health services and quickly resume optimal performance, particularly in scenarios involving treatment restrictions or limitations (Italy in 2020 and 2022 55 and Belgium in 2023; 35 Table 3).Resilience was defined in relation to the COVID-19 pandemic.Italian regions assessed their capacity to sustain the same level of urgent treatments and follow-up visits to avert potential unmet needs, 55 while Belgium examined the number of new invasive cancer cases as a direct adverse impact on the health of the population. 35This operationalization covers the first two aspects of the definition of resilience proposed by the EU's expert panel on effective ways of investing in health, namely sustaining required operations and resuming optimal performance. 11

Protect vulnerable groups
Belgium (2023), 35 Estonia (2023) 42 and Italy (2021) 55 introduced resilience measures aimed at reducing vulnerabilities to better withstand current and future shocks (Table 3).These countries considered key performance indicators that evaluated reducing health system vulnerability through COVID-19 vaccination coverage.Some national health system performance assessments broadened this concept by including indicators to assess the health system's ability to protect vulnerable groups through preventive efforts.Estonia (2023) 42 incorporated indicators measuring seasonal vaccination coverage rates and the incidence of vaccine-preventable diseases per 100 000 population.Italy (2022) 55 integrated metrics of influenza vaccination coverage for older people and health workers.Czechia (2023) 41 and Italy (2022) 55 developed measures that gauge the health system's capacity to provide mental health services.This operationalization overlaps with the last category of the EU's expert panel, that is, reducing vulnerability to future shocks. 11

Acquire and use resources
A more nuanced conceptualization of resilience emerged in some countries which placed greater emphasis on structure-related elements, which is consistent with the EU's 2016 and 2018 interpretation of resilience. 72,73ey performance indicators included generic pharmaceutical usage, public health, long-term care expenses and bed occupancy rates (Belgium, 2023); 35 and average waiting time between tests and results and surge capacity (Ireland, 2023). 54Croatia (2023; Albreht et al., unpublished report, available on request from the corresponding author) and Czechia (2023) 41 focused on the enhancement of investment and the promotion of policy reform, with the aim of strengthening technological and infrastructure capabilities (Table 3).This operationalization overlaps with the third aspect of the definition of resilience of the EU's expert panel, namely transforming health system structure and functions to enhance the system's strength. 11

Trained and prepared staff
Resilience can also be assessed through measures related to the health-care workforce, as their dedication and well-being are integral to maintaining resilient health-care systems.Indicators can be either quantitative, such as the number of medical doctors and nurses or beds (Belgium 2023, 35 Czechia 2023 41 and Ireland 2023); 54 or qualitative, which focuses on well-being of health professionals.This latter aspect involves evaluating job satisfaction through metrics such as intention to leave and absenteeism, as adopted by Belgium (2023), 35 Croatia (2023) 39 and Ireland (2023). 54Ireland (2023) 54 incorporated key performance indicators to monitor specific policies on human resources such as establishing helplines for professionals (Table 3).This operationalization crosses two dimensions of the definition of resilience of the EU's expert panel, that is, sustaining required operations and transforming health-care structure and functions to enhance the system's strength. 11

Utilize digital health
Some countries have incorporated digital channels into their resilience dimension, driven by the accelerated digitalization of health care in response to the COVID-19 pandemic.Belgium in 2023 35 and Italy in 2021 and 2022 55 integrated these supply-side indicators into the health system performance assessments to monitor whether healthcare systems succeeded in reducing backlogs and ensuring continuity of care by delivering services through digital channels (Table 3).This operationalization crosses two dimensions Fig. 1

Strengthen primary health care
Resilience is sometimes construed as the capacity to strengthen primary care as a form of preparedness.Significant links existed between the core functions of primary care in non-health emergencies and a country's ability to effectively respond to and recover from the COVID-19 pandemic. 3Some countries incorporated indicators of primary health care into the resilience dimension to highlight the correlation between the capacity to provide care during outbreaks.Indicators related to primary health care often relate to investments in structural aspects, such as long-term care (e.g.Croatia in 2023, 39 Czechia in 2023 41 and Italy in 2022). 55Some indicators emphasize the reinforcement of access, such as the number of contacts with general practitioners and mental health services provided (e.g.Czechia in 2023 41 and Italy in 2022); 55 and vaccination coverage for vulnerable populations (Italy, 2021 and 2022), 55 which in some instances are included in other dimensions (Belgium, 2023). 35In a broader context, all the health system performance assessments examined incorporated primary health-care indicators either within specific domains or as cross-cutting factors.These indicators assess the effectiveness of primary health care through measures such as ambulatory-related conditions, access, coordination and service continuity.Some countries included primary health-care indicators within other dimensions directly tied to resilience, such as expanding the primary healthcare workforce and allocating health expenditure to primary care (Ireland 54 and Croatia 39 in 2023).Additionally, the catchment index (i.e. the number of visits or diagnostic examinations in relation to those prescribed) contributes to resilience by revealing potential unmet needs (Table 3).This holistic vision of primary health care aligns with the aim of resilience to ensure systems can bounce back, adapt, learn and improve in crises, potentially spanning all dimensions of resilience as defined by the EU's expert panel on effective ways of investing in health.

Discussion
We investigated how resilience has been defined and integrated into various European health system performance assessment frameworks.We sought to clarify the conceptual framing of resilience by examining the key performance indicators in these health system performance assessments.Although the EU recommended including resilience in health system performance assess-ments as early as 2014, 27 all the countries analysed in our study only incorporated this dimension after the COVID-19 pandemic.
Resilience refers to a health system's capacity to adapt and maintain control over its structure and functions, even when confronted with significant stresses. 74Traditionally, the focus has been on risk-management strategies to prevent and mitigate threats, but the complexities of contemporary systems make this approach insufficient. 75The COVID-19 pandemic led to a paradigm shift that acknowledged the unpredictability of systemic threats and emphasized the need to enhance health system resilience. 76This perspective highlights the importance of a health system's ability to anticipate, absorb, recover from and adapt to a wide range of disruptions. 76,77

Country
First year of assessment Incorporating resilience as an element within national and regional health system performance assessments 78 is a practical strategy to enhance the ability of health systems to withstand and recover from disruptions efficiently.The findings of a recent analysis of health system performance assessments 7 have been partially integrated by most national health system performance assessments, demonstrating the ability of measurement systems to adapt to contemporary environments.This adaptation is particularly evident in the case of Italy, where resilience indicators dominated during the pandemic, both in system delivery (e.g.testing for COVID-19 and COVID-19 vaccination) and final goals of the health system (e.g.pandemic mortality rate).In contrast, in non-pandemic periods, key performance indicators of resilience mainly related to primary health care, particularly in the domains of system delivery and intermediary outcomes.This finding aligns with the broader understanding that resilient health-care systems require robust primary healthcare foundations 7 to ensure accessibility, equity and continuity of care, even in the face of unprecedented challenges. 79,80he concept of resilience seems to have evolved, as shown by the changes in the key performance indicators within the Belgian and Italian health system performance assessments.This shift underscores the importance of adaptability in implementing resilience strategies in response to changing environments. 56verall, most countries have adopted a definition of health system resilience that emphasizes the ability to anticipate, absorb and adapt to shocks through the following dimensions: (i) capacity to address unmet needs and maintain outcomes; (ii) capacity to protect vulnerable groups; (iii) capacity of management to acquire and use resources; (iv) capacity to have trained and prepared staff in place; (v) capacity to utilize digital health; and (vi) capacity to strengthen primary health-care services.However, each country has tailored this definition to suit its unique health-care landscape and lessons learnt from the pandemic.Some countries have prioritized maintaining essential health services and quickly resuming optimal performance during a pandemic, while others have focused on reducing vulnerabilities within their health-care system.Our analysis indi-cates that health system performance assessment frameworks incorporating resilience emerged in updated postpandemic assessments.Initially, countries with pre-existing health system performance assessments introduced resilience metrics in response to the COVID-19 pandemic with a focus on maintaining essential services during critical phases.Later, these countries assessed the resilience of their health-care systems to future shocks by introducing key performance indicators related to the health workforce, digital health and strengthening of primary health care.In particular, investments in digital technologies, such as digital consultations and telehealth services, can streamline

Research
patient pathways, minimizing the need for in-person doctor visits. 3While vital for resilience of health-care systems, these investments necessitate integrated information systems and care models to enhance patient-care coordination and decision-making.The concept of resilience has been addressed to a lesser extent in wider health system performance assessments.This concept has evolved from addressing immediate outbreak responses dur-ing the initial phases of a pandemic, to a stronger focus on proactive preparedness measures aimed at mitigating future epidemic impacts through strengthening health-care systems.The implementation and scaling up of these measures depends on the availability of data.Some data, such as using digital health and strengthening primary health care, could be easily collected and included in health system performance assessments.However, other data, such as prepared-ness measures, are not systematically monitored.Finally, while some key performance indicators may have a broader scope, such as job satisfaction of health workers, they can be classified under resilience.The inclusion of such key indicators as a measure of resilience was identified in the case studies analysed as a predictor of health-care systems' readiness for future shocks.
Our study has some limitations.First, our inclusion criteria were restricted to health system performance assessments explicitly mentioning the term resilience.Countries using different terminologies, such as preparedness and responsiveness, were not included.However, the objective of our analysis was to examine resilience and how it was defined and operationalized into key performance indicators.Future studies could investigate how countries adopted different terms to refer to the concept of resilience.Additionally, we limited the geographical scope to the EU, for reasons outlined in the methods section.Nevertheless, it is important to acknowledge global heterogeneity in health system performance assessment frameworks and to provide perspectives beyond Europe.Thus, the study may only partially capture the diversity in health system performance assessments globally.■ Bull World Health Organ 2024;102:498-508| doi: http://dx.doi.org/10.2471/BLT.23